Device,method and system for multimedia content adaptation

ABSTRACT

The invention concerns a device receiving data associated with a first profile and generating data associated with a second profile, comprising means to extract facts associated with the first profile, means to extract facts associated with the second profile, a set of rules describing a way to modify facts associated with said first profile into facts associated with said second profile, means to analyze said facts associated with the first profile and said facts associated with the second profile, means to modify facts associated with said first profile into facts associated with said second profile according to said set of rules. The device is characterized in that it comprises, means to request additional rules when said set of rules does not allow to modify facts associated with said first profile into facts associated with said second profile.

[0001] The invention concerns a device and a method to adapt multimediacontent to different terminals.

[0002] More particularly, the invention allows the dynamic adaptation ofmultimedia content to different terminal profiles.

[0003] The invention is particularly interesting in the MPEG-4 domainwhere a certain number of profiles are defined according to differentterminals capabilities.

[0004] In the context of multimedia design, several kinds ofheterogeneous terminals are supposed to be able to receive and recovercontent provided by the same server.

[0005] However, the terminals have different capabilities, in terms ofhardware or software functionalities. These terminals can be desktoppersonal computers, laptop computers, personal digital assistant (PDA),portable phones, or digital television receivers. As it is easilyunderstandable and well known, these terminals cannot read the samecontent. As a simple example, one can easily understand that theseterminals cannot each display a same image with the same resolution.

[0006] The MPEG-4 standard (defined in the document ISO/IEC 14496-1:2001Systems) consortium has defined a number of profiles according todevices that have to be MPEG-4 compliant. The profiles are defined inthe document N3932 “New MPEG-4 Profiles Under Consideration”.

[0007] For example, some PDAs will be able to play only MPEG-4compressed audio/video contents and some MPEG-4 compliant computers willbe able to play all MPEG-4 profiles.

[0008] The problem of MPEG-4 content creators is to design content thatis understood by platforms that do not comply with the profile of thereceived content. A content created for some specific MPEG-4 profilescannot be displayed on platforms that do not comply with the content'sprofiles or at least not on those that comply with a profile of a lowerlevel. Moreover, the number of defined profiles is quite important aseach profile is also split into different levels, each level definingparameters such as visual resolution or a data rate for instance.

[0009] The document “Design of a framework for dynamic contentadaptation to web-enabled terminals and enterprise applications”published in the proceedings of the Sixth Asia Pacific SoftwareEngineering Conference pages 72 to 79 concerns a method to adapt contentto terminals having different capabilities. A solution that selects,deletes or converts each element of a web document's contentsappropriately according to the type of devices is proposed. Despite thefact that the document describes a dynamic adaptation system, theadaptation is carried out at the transmitter, resulting in manydifferent versions of the content to be transmitted.

[0010] This solution is not adapted to a system comprising a greatnumber of terminals, where it is desirable to send a same content to thelargest possible number of terminals, or a system in which thetransmitter has no advance knowledge of parameters stored in aparticular terminal.

[0011] The inventors have found that it should be interesting to have asystem that allows a dynamic content adaptation in multicasttransmissions, when several predefined contents are generated accordingto different profiles and delivered to different heterogeneous terminalsthat are not compliant to the profile of the delivered content.

[0012] The invention proposes a solution based on conversion rules, andthat allows modifying the rules dynamically. This solution isparticularly interesting in systems where a great number of profiles aredefined and where loading, by a terminal, of a complete solution thatallows to convert all profiles in each terminal is costly in terms ofresource consumption and cost.

[0013] The object of the invention is a device for multimedia contentadaptation receiving input data associated with at least a first profileand generating output data associated with at least a second profile.The device comprises:

[0014] means to extract facts associated with at least a first profile,

[0015] means to extract facts associated with at least a second profile,

[0016] a set of rules describing a way to modify facts associated withsaid first profile into facts associated with said second profile,

[0017] means to analyze said facts associated with at least a firstprofile and said facts associated with at least a second profile,

[0018] means to modify facts associated with said first profile intofacts associated with said second profile according to said set ofrules,

[0019] means to request additional rules when said set of rules does notallow to modify facts associated with said first profile into factsassociated with said second profile according to said set of rules.

[0020] Having a set of rules that can be modified according to thereceived content and to the profile of the terminal reduces the requiredmemory capacity in the terminal. The invention is particularlyinteresting in—but not limited to—a system allowing different profiles,each profile being split into several levels. As a function of thereceived content, new rules are generated that will allow the conversionof more complex functions in case this should be required. Thanks tothis interactivity between the receiving terminal and the outside, it ispossible to enrich the set of rules with new rules and thus to convertall the functions, even specific facts that are not necessarily oftenused and that would not require permanent storage of dedicated rules.

[0021] The invention can be used in—but is not limited to—networksconnecting several multimedia terminals, which generate several contentsaccording to different profiles.

[0022] In a preferred embodiment, the device includes means to storesaid set of rules and said additional rules.

[0023] According to an embodiment of the invention, it is proposed touse a system able to learn new rules when the existing rules are notsufficient, e.g. an expert system.

[0024] This feature concerns particularly devices having little memorycapacity such as portable phones or personal digital assistants. In thiscase, a set of basic rules can be first implemented in the device and ifit is not sufficient to make the conversion, additional rules arerequested.

[0025] This is particularly interesting if a device receives regularlycontent written according to a particular profile. The set of rulesnecessary for adapting the content being in this case often used, it istime saving to maintain the appropriate rules in the device for futureuse.

[0026] In another embodiment, said set of rules and said additionalrules are received with the content.

[0027] This is particularly suitable for devices having no storagecapacity to permanently store said set of rules and said additionalrules.

[0028] According to an embodiment of the invention, the means to requestadditional rules send a request to a user.

[0029] In case new profiles are defined or multiple adaptation solutionsexist, it is interesting to enable the user to have the possibility tochoose among the multiple conversion possibilities. According to theusers' preferences, for instance, he can prefer having a polygonconverted in a circle rather than converted into a rectangle. Thepreferences of the user are rules that may help the conversion of futurecontents and therefore can be memorized in the device.

[0030] According to an embodiment of the invention, the device accordingto the invention is connected to one multimedia terminal.

[0031] Connecting the devices to multimedia terminals is particularlyadvantageous. Preferentially, the device can be fully part of themultimedia terminal instead of being connected to it through an externallink.

[0032] According to an embodiment of the invention, said multimediaterminal is linked to a server through a network and said server sendssaid input data according to a plurality of profiles, and said means torequest additional rules sends a request to said server.

[0033] The invention is particularly interesting in such a client-serverenvironment where a server transmits the data to all the clientterminals.

[0034] Preferentially, said additional rules depend on the currentprofile of said multimedia terminal.

[0035] The device can request additional rules in order to complete theset of rules already stored in its memory and enabling to convert thefacts associated with the profile of the incoming data into factsassociated with its profile. A terminal can be for instance configuredaccording to one profile at one instant and its configuration can bechanged according to network characteristics or other external orinternal parameters. So, according to this change of configuration, theterminal can need other rules for the adaptation to this new profile.

[0036] In a particular embodiment, said set of rules and said additionalrules are classified according to a priority order.

[0037] In some cases, it can happen that several rules enable theconversion of the facts associated with the first profile into factsassociated with the second profile. In this case, it is particularlyconvenient to decide which rule must be applied. The rules can beclassified by the user. The classification can also be based onstatistics of the user's last choices.

[0038] According to a particular embodiment of the invention, theprofiles are defined according to MPEG-4 profiles definition.

[0039] In MPEG-4, profiles are strictly defined for each part of thestandard:

[0040] Video

[0041] Audio,

[0042] Graphics

[0043] Scene graph

[0044] DMIF (standing for <<Delivery Multimedia Integration Framework>>)

[0045] OD (standing for “Object Descriptor”)

[0046] MPEG-J

[0047] Each profile defines a set of functionalities that should besufficient for a certain applicative purpose, for instance for video ondemand or quality of service requirements.

[0048] In MPEG-4, the profiles are also split into several levels.Levels define the complexity of profiles. Combination of profiles andlevels define a device performance (processing power, graphicscapabilities, memory for instance).

[0049] The invention concerns also a system comprising at least onedevice according to any of the embodiments of the invention, one firstdevice generating an output content compliant to said first profile andsent to at least one second device that can understand an input contentcompliant to a said second profile.

[0050] According to the invention, the device according to anyembodiment of the invention receives said output content compliant tosaid first profile and generates an output content compliant with a saidsecond profile for said second device.

[0051] The invention relates also to a server for multimedia contentadaptation comprising means to send data to at least one clientterminal, said data being associated with at least a first profile andsaid client terminal being compliant to at least a second profile, saidclient terminal comprising a set of rules for converting said dataassociated to a first profile into data associated to said secondprofile.

[0052] According to the invention, the server comprises also means tosend additional rules to at least one of the client terminal uponrequest of said client terminal when said client terminal can notconvert said data associated to the first profile into data associatedto the second profile.

[0053] The invention relates also to a method for multimedia contentadaptation comprising the steps of

[0054] receiving input data associated with a first profile,

[0055] generating output data associated with at least a second profile,

[0056] extracting facts associated with the first profile,

[0057] extracting facts associated with the second profile,

[0058] describing, with a set of rules, a process for modifying factsassociated with the first profile into facts associated with the secondprofile,

[0059] analyzing said facts associated with the first profile and saidfacts associated with the second profile,

[0060] modifying facts associated with said first profile into factsassociated with said second profile according to the set of rules.

[0061] requesting additional rules when the set of rules does not allowmodifying facts associated with said first profile into facts associatedwith said second profile according to said set of rules.

[0062] The invention relates also to a computer program productcomprising program code instructions to execute the method according tothe invention when this program is executed on a set-top box or on adigital television receiver.

[0063] Other characteristics and advantages of the invention will appearthrough the description of a non-limiting embodiment of the invention,which will be illustrated, with the help of the enclosed drawings amongwhich:

[0064]FIG. 1 is a block diagram representing a device according to theembodiment of invention,

[0065]FIG. 2 is a block diagram describing the adaptation in case ofcontent compliant to the MPEG-4 standard,

[0066]FIG. 3 is a block diagram representing a system according to aparticular embodiment of the invention.

[0067] The present embodiment uses MPEG-4 profiles, but other types ofprofiles may also be used. When the term profile is used in thedescription, this also includes the notion of level.

[0068] The recent definition of profiles in the MPEG-4 consortium hasstressed the idea that despite a single content is generated, a lot ofheterogeneous terminals should be able to read it as more and more theuse of multimedia content has to be read by several terminals.Therefore, the invention is particularly relevant when applied toMPEG-4.

[0069] A device according to the present embodiment is represented onFIG. 1. The device includes a content loading module 1 connected to ananalysis and adaptation system 3 and to an external link 6. The analysisand adaptation system 3 is also connected to a rules storage means 2 andto an output module 4. The rules storage means enable a permanentstorage of the rules. The rules storage means 2 is optional and inanother embodiment the rules can be transmitted with the content asdescribed later in this document. and deleted once used.

[0070] In a preferred embodiment, as represented, the adaptation moduleis also connected to the user through a connection link 5.

[0071] According to the present embodiment, the external link 5 is awireless remote control, while the device is part of a televisionreceiver. Of course, the invention is not limited to this environment.The devices are preferentially included in electronic devices such aspersonal computers, personal digital assistants, television sets or anyother multimedia terminal and these multimedia terminals can beconnected.

[0072] The device receives content from the external link 6. Accordingto the present embodiment, this link is an Internet connection.

[0073] In the preferred embodiment described here, the content is MPEG-4compliant. The content is written according to the different profilesdefined by the MPEG-4 standardization committee. Each profile is alsosplit into several levels.

[0074] A server, as shown on FIG. 3, connected to several devices,generates the content.

[0075] According to a variant embodiment, the content is generated by adevice and broadcast by the server to other devices of the network.

[0076] The generated content complies with one profile, which is notnecessarily the profile of all the addressees of the content and thusmust be converted in order to be understood and/or displayed by eachaddressee.

[0077] The content loading module 1 may have also to decode the content(e.g. when it is compressed). The decoding consists for instance inseparating the different components of the MPEG-4 scene into the video,the audio, the graphics elements.

[0078] The content loading module transmits the content to the analysisand adaptation system, which splits it into several components,corresponding to the profile tools (audio, video , . . . ). An example,in the case of an MPEG-4 content, is given by FIG. 2.

[0079] Once the content is split, the analysis and adaptation moduleextracts facts from this modified content in order to apply the rulesstored in the storage module 2. These rules enable to convert thecontent defined according to the initial profile into a content that canbe understood by the television receiver into which the device isembedded. The profile can be defined as a set of facts such as the sizeof the display if it is a content that has to be displayed. Forinstance, facts provided by the terminal can be the size of the display,the number of colours it can display, the resolution etc . . .

[0080] The rules stored in the rules storage means 2 enable to modifythe content by modifying the different facts defined and converting theminto facts corresponding to the terminal profile.

[0081] A simple rule can be for instance:

[0082] If terminal size is 50 * 10 and the input content is defined as100 * 20, apply a scale of ½ on original content.

[0083] The adaptation module may have multiple choices depending on thefacts extracted from the input content. According to a variantembodiment, rules are ranked according to a priority criterion. Forinstance, if at one time a user's opinion is requested to select a ruleamong a plurality of rules, this user preference is memorized and therules are classified according to statistics based on the user'sprevious choices.

[0084] According to a variant embodiment, additionally, internal rulesare added for optimisation purpose, for instance, in MPEG-4, “removingor deleting empty nodes”. The nodes describe audio-visual primitives andtheir attributes, along with the description of the scene graph itself.

[0085] According to a variant embodiment, some rules are deleted in thedevice according to the profile of the terminal associated with thedevice according to the invention. Indeed, some rules can be useless ifthe terminal never receives content of a given profile. This can beinteresting for low profile terminals, which are generally associatedwith low software or hardware capabilities and may therefore have littlememory.

[0086] According to the content received, some of the extracted factsmay not have any corresponding fact according to the terminal's profile.This can happen in the case of a content defined according to a highlevel profile and a terminal complying with a low level profile, havinglittle resources.

[0087] In this case, the rules storage means may not contain conversionrules that could convert facts according to the high level profile.Also, a previously present rule may have been deleted.

[0088] According to the present embodiment, the missing rules arerequested by the device through the external link 6 from the server. Theserver responds by sending back the rules to the device.

[0089] When the device receives an incoming content, it converts thiscontent into the different objects, as explained on FIG. 2. It extractsthe different facts from these objects. Then, it puts in accordance withthese facts the facts associated with the output profile. When there isno fact that can be put in accordance, the device cannot find any ruleto make the conversion. It requests then to the server a rule, or a setof rules, that enable the conversion. This request is transparent forthe user and real time.

[0090] According to a variant embodiment, the rules are requested by thedevice from the user through the external link 5. The user can then helpthe adaptation module with the conversion by entering an indication tothe device. The device proposes for instance several choices to the userand the user makes the choice. Then the choice is converted into a rulesuch as “convert a polygon into a rectangular”.

[0091] It means that according to different embodiments, either link 5or link 6 can be available.

[0092] On FIG. 2, a decomposition of an MPEG-4 content is represented aswell as the functional description of the adaptor.

[0093] The content is separated into the different elements, audio,video, graphics, scene graph, object descriptors and MPEG-J.

[0094] The adaptor includes at least 4 modules: a module 14 for temporaland spatial resampling, a module 15 for scene reconstruction, a module16 for object descriptor reconstruction and a module 17 for API(acronyme anglais de “application and programming interface significant“interface pour la programmation d'applications”) verification.

[0095] The module 14 of temporal and spatial re-sampling adapts forinstance the size or the shape (or any other geometrical feature of theinput content extracted from the input facts to the output profileaccording to the facts. For instance, let's suppose that the inputcontent contains a polygonal shape and the output profile contains factsindicating that any-authorized shape has to be rectangular.

[0096] The temporal and spatial re-sampling module 14 adapts the size ofa video stream or its frame rate, depending on the input facts and factsof the output profile.

[0097] The scene reconstruction sub-module 15 reconstructs a scene thatis coherent with the input scene but adapted to the output Scenegraphand graphics profiles. The output Scenegraph represents the architectureof the scene.

[0098] For instance, if the input content contains a polygonal shape andthe output profile contains facts such as authorized shape isrectangular, the module 15 defines a rule that converts a polygonalshape into a rectangular shape.

[0099] The Object descriptor reconstruction module 16 provides the gluebetween the scene description provided by the scene reconstructionsub-module 15 and the streaming resources.

[0100] The API verification module 17 checks that the content iscompliant with the output profile.

[0101] According to this embodiment of the invention, the differentfunctionalities included in the analysis and adaptation system 3 areexecuted as computer program instructions.

[0102] The set of rules included in the rule storage means 2 can bemodified and therefore can be upgraded without the need of acompilation. They can be downloaded as software can be or they can bemodified each time a new rule is added upon request of the device to theserver. This enables a dynamic adaptation not dedicated to one terminalbut that can be adapted to every terminal including such a device.

[0103] On FIG. 3, a system including several devices according to apresent embodiment is represented.

[0104] This figure represents a particular embodiment where severalnetworks are inter-connected through several gateways and connected tothe same server.

[0105] This specific embodiment shows the devices distributed in thenetworks instead of integrated into other apparatus. This allowsimplementing the invention using existing terminals. One particularlyinteresting configuration is to group the terminals according to theirprofiles on the same network. In this case, a server 7 sends one contentdesigned according to one profile on a network 8. The content isreceived by the different gateways or relays 9, 10 and transmitted tothe addressees connected to the different networks 11, 12, and 13. Aspreviously said, if the terminals connected to the network 11 complywith a unique profile, it is particularly interesting to have theadapter located in the gateway, so that the adapter makes the conversiononly once for all the terminals. The adapter, in this case, can learneasily the type of profile the terminals located on the network complywith and learn the appropriate rules. The adapter can contain an expertsystem able to learn new rules. One condition for sending the samecontent for all the terminals in this case is that all the users if theyare asked for it, give the same information to the adapter. This seemsto be unrealistic. According to the present embodiment, a single masterterminal is designed to help the adapter with the conversion, only ifnecessary. In a preferred embodiment, in the case of an IEEE-1394network, the master terminal can be for instance the cycle masterterminal.

[0106] According to a variant embodiment, the rules (base rules andadditional rules) are sent with the content. This is particularlysuitable for adapters having no memory. The server can send the samecontent to the different adapters 9 and 10 and send to these adaptersdifferent rules according to the terminals on the networks 11, 12 and13.

[0107] Advantageously, the content creator can also define a specificset of rules dedicated to the type of content he creates and this set ofrules is sent with the content.

1. Device for multimedia content adaptation receiving input dataassociated with a first profile and generating output data associatedwith a second profile, comprising means to extract facts associated witha first profile, means to extract facts associated with a secondprofile, a set of rules describing a way to modify the input dataaccording to the facts associated with said first profile and to thefacts associated with said second profile, means to modify the inputdata to fit with the facts associated with said second profile accordingto said set of rules, wherein it comprises means to request additionalrules when said set of rules does not allow to modify the input data tofit with the facts associated with said second profile according to saidset of rules.
 2. Device according to claim 1 wherein it includes meansto store said set of rules and said additional rules.
 3. Deviceaccording to claim 1 wherein said set of rules and said additional rulesare received with the input data.
 4. Device according to claim 1 whereinsaid means to request additional rules send a request to a user. 5.Device according to claim 1 wherein it is connected to one multimediaterminal.
 6. Device according to claim 5 wherein said multimediaterminal is linked to a server through a network, said server sends saidinput data according to a plurality of profiles, and said means torequest additional rules sends a request to said server.
 7. Deviceaccording to claim 6 wherein said additional rules depend on the currentprofile of said multimedia terminal.
 8. Device according to claim 1wherein said set of rules and said additional rules are classifiedaccording to a priority order.
 9. Device according to claim 1 whereinsaid profiles are defined according to MPEG-4 profiles definition. 10.System comprising one first device which generates an output contentcompliant with a first profile and sent to at least one second devicethat can understand an input content compliant to a said second profile,wherein it comprises a device according to claim 1 which receives saidoutput content compliant with said first profile and generates an outputcontent compliant with said second profile for said second device. 11.Method for multimedia content adaptation comprising the steps ofreceiving input data associated with a first profile, generating outputdata associated with a second profile, extracting facts associated withfirst profile, extracting facts associated with the second profile,describing, with a set of rules, a process for modifying the input dataaccording to the facts associated with said first profile and to thefacts associated with the second profile, modifying the input data tofit with the facts associated with said second profile according to theset of rules, wherein it comprises the step of requesting additionalrules when the said set of rules does not allow modifying factsassociated with said first profile into facts associated with saidsecond profile according to said set of rules.
 12. Server for multimediacontent adaptation comprising means to send data to at least one clientterminal, said data being associated with at least a first profile andsaid client terminal being compliant to at least a second profile, saidclient terminal comprising a set of rules for converting said dataassociated to a first profile into data associated to said secondprofile, said server being wherein it comprises means to send additionalrules to at least one of the client terminal upon request of said clientterminal when said client terminal can not convert said data associatedwith the first profile into data associated with the second profile. 13.Computer program product comprising program code instructions to executethe method according to claim 11 when this program is executed on aset-top box or on a digital television receiver.